POSET PINBALL, THE DIMENSION PAIR ALGORITHM, AND TYPE A REGULAR NILPOTENT 

HESSENBERG VARIETIES 



DARIUS B AYEG AN AND MEGUMI HARADA 

Abstract. In this manuscript we develop the theory of poset pinball, a combinatorial game recently introduced 
by Harada and Tymoczko for the study of the equivariant cohomology rings of GKM-compatible subspaces of 
GKM spaces. Harada and Tymoczko also prove that in certain circumstances, a successful outcome of Betti 
poset pinball yields a module basis for the equivariant cohomology ring of the GKM-compatible subspace. Our 
main contributions are twofold. First we construct an algorithm (which we call the dimension pair algorithm) 
which yields the result of a successful outcome of Betti poset pinball for any type A regular nilpotent Hessen- 
berg and any type A nilpotent Springer variety, considered as GKM-compatible subspaces of the flag variety 
3'£ags(C"). The definition of the algorithm is motivated by a correspondence between Hessenberg affine cells 
and certain Schubert polynomials which we learned from Erik Insko. Second, in the special case of the type A 
regular nilpotent Hessenberg varieties specified by the Hessenberg frmction = h{2) = 3 and h{i) = i + 1 for 
3 < j < n — 1 and h{n) = n, we prove that the pinball result coming from the dimension pair algorithm is poset- 
upper-triangular; by results of Harada and Tymoczko this implies the corresponding equivariant cohomology 
classes form a H*^ (pt)-module basis for the S'^-equivariant cohomology ring of the Hessenberg variety. 
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1. Introduction 

The purpose of this manuscript is to further develop the theory of poset pinball, a combinatorial game 
introduced in [10] for the purpose of computing in equivariant cohomology ringsElin certain cases of type 
A nilpotent Hessenberg varieties. One of the main uses of poset pinball in IITOl is to construct module 
bases for the equivariant cohomology rings of GKM-compatible subspaces of GKM spaces IITOl Definition 
4.5]. In the context of this manuscript, the ambient GKM space is the flag variety 3^£ags(C") equipped with 
the action of the diagonal subgroup T of U{n, C), and the GKM-compatible subspaces are the nilpotent 
Hessenberg varieties. It is well-recorded in the literature (e.g. flHl and references therein) that GKM spaces 
often have geometrically and /or combinatorially natural module bases for their equivariant cohomology 
rings; the basis of equivariant Schubert classes {crw}weSn fo^" H^{3^£ags{C"')) is a famous example. The 
results of this manuscript represent first steps towards the larger goal of using poset pinball to construct 
a similarly computationally effective and convenient module bases for a GKM-compatible subspace by 
exploiting the structure of the ambient GKM space. 
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We briefly recall the setting of our results. Let iV : C" C" be a nilpotent operator. Let /i : {1, 2, . . . , n} — > 
{1, 2, . . . , n} be a function satisfying h{i) > i for all 1 < i < n and h{i + 1) > h{i) for all 1 < i < n. The 
associated Hessenberg variety Hcss(iV, h) is then defined as the following subvariety of 3^iags{<C^): 

Hcss(7V, /i) := {F, = (0 C V^i C C • • ■ C K-i Q K = C") | NV^ C Vh(i) for all z = 1, ... , n}. 

Since we deal exclusively with type A in this paper, henceforth we omit this phrase from our terminology. 
Two special cases of Hessenberg varieties are of particular interest in this manuscript: when N is the princi- 
pal nilpotent operator (in this case Hess(iV, h) is called a regular nilpotent Hessenberg variety) and when 
h is the identity function h{i) — i for all 1 < i < n (in this case Hess(A^, h) is called a nilpotent Springer 
variety and is sometimes denoted Sat). Hessenberg varieties arise in many areas of mathematics, including 
geometric representation theory ll9l [T5llT6 ^, numerical analysis f6], mathematical physics fl2llT4l , combi- 
natorics |8|, and algebraic geometry ||4l|5l, so it is of interest to explicitly analyze their topology, e.g. the 
structure of their (equivariant) cohomology rings. We do so through poset pinball and Schubert calculus 
techniques, as initiated and developed in |[Tl lTOl[T]l | and briefly recalled below. 

The following relationship between two group actions on the nilpotent Hessenberg variety and the flag 
variety respectively allows us to use the theory of GKM-compatible subspaces and poset pinball. There is 
a natural subgroup of the unitary diagonal matrices T which acts on Hcss(iV, h) (defined precisely in 
Section|2]l. The group T, the maximal torus of U{n, C), acts on J'£ags{C") in the standard fashion. It turns 
out that the 5^-fixed points Hess(iV, h)^^ are a subset of the T-fixed points 'Jlags{C^)^ = 5„. Moreover, the 
inclusion of Hcss(A^, h) into ?'€ags(C") and the inclusion of groups S*^ into T then induces a natural ring 
homomorphism 

(1.1) H^{jeags{C")) H*s^ (Hess(iV, h}}. 

As mentioned above, it is well-known in Schubert calculus that the equivariant Schubert classes {a-w}weSn 
are a computationally convenient _ff^(pt) -module basis for H^{J'£ags{C")). We refer to the images in 
Hgi (Hcss(iV, h)) of the equivariant Schubert classes {crw}-wes„ via the projection l|l.Hl as Hessenberg Schu- 
bert classes. Given this setup and following lilOI , the game of poset pinball uses the data of the fixed points 
J£ags{C")'^ = S„ (considered as a partially ordered set with respect to Bruhat order) and the subset 

Hess(iV, hf' C jeags{C"f ^ Sn 

to determine a set of roUdowns in Sn ■ It is shown in flOl that, imder certain circumstances (one of which is 
discussed in more detail below), such a set of rolldowns in turn specifies a subset of the Hessenberg Schu- 
bert classes which form a i/^i (pt)-module basis of i/^i (Hess(A^, h)). Thus poset pinball is an important 
tool for building computationally effective module bases for the equivariant cohomology of Hessenberg 
varieties. Indeed, the results of [llj accomplish precisely this goal - i.e. of constructing a module basis via 
poset pinball techniques - in the special case of the Peterson variety, which is the regular nilpotent Hessen- 
berg variety with Hessenberg function h defined by h{i) = i + 1 for 1 < i < n — 1 and h{n) ~ n. Exploiting 
this explicit module basis, in [TTl Theorem 6.12] the second author and Tymoczko give a manifestly positive 
Monk formula for the product of a degree-2 Peterson Schubert class with an arbitrary Peterson Schubert 
class, expressed as a H^i (pt)-linear combination of Peterson Schubert classes. This is an example of equi- 
variant Schubert calculus in the realm of Hessenberg varieties, and it is an open problem to generalize the 
results of | | 11 | to a wider class of Hessenberg varieties. 

We now describe our main results. First, we explain in detail an algorithm which we dub the dimen- 
sion pair algorithm and which associates to each S'^-fixed point w G Hcss(A^, h)^ a permutation in Sn, 
which we call the roUdown of w following terminology in flO\ and denoted roi£{w) 6 Sn- In the spe- 
cial cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties, we show that the set 
{roW(u;)}^gjj^ggj^ ^^si can be interpreted as the result of a successful game of Betti pinball (in the sense 
of Hop . The main motivation for our construction is that a successful outcome of Betti pinball can, under 
some circumstances, produce a module basis for the associated equivariant cohomology ring (cf. flO. Sec- 
tion 4.3]). In this sense, our algorithm represents a significant step towards the construction of module bases 
for the equivariant cohomology rings of general nilpotent Hessenberg varieties, thus extending the theory 
developed in [10 11]. Although we formulate our algorithm in terms of dimension pairs and permissible 
fillings following terminology of Mbirika ||T31 , the essential idea comes from a correspondence between 
Hessenberg affine cells and certain Schubert polynomials which we learned from Erik Insko. 
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Second, for a specific case of a regular nilpotent Hessenberg variety which we call a 334-type Hessen- 
berg variety, we prove that the set of rolldowns {foU{'w)}^^^^^^f^j^ ^^si obtained from the dimension pair 
algorithm is in fact poset-upper-triangular in the sense of |T0). As shown in flOj, this is one of the possi- 
ble circumstances under which we can conclude that the corresponding set of Hessenberg Schubert classes 
forms a module basis for the 5 ^-equi variant cohomology ring of the variety. Thus our result gives rise to a 
new family of examples of Hessenberg varieties (and GKM-compatible subspaces) for which poset pinball 
successfully produces explicit module bases. We mention that the dimension pair algorithm also produces 
module bases in a special case of Springer varieties Q- Although we do not know whether the dimension 
pair algorithm always succeeds in producing module bases for the S'^-equivariant cohomology rings for a 
general nilpotent Hessenberg variety, the evidence thus far is suggestive. We leave further investigation to 
future work. 

We give a brief summary of the contents of this manuscript. In Section |2] we recall some definitions and 
constructions necessary for later statements. In Section ISD we describe the dimension pair algorithm and 
prove that the result of the algorithm satisfies the conditions to be the outcome of a successful game of Betti 
poset pinball in the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer varieties. 
We briefly review in Section [3l2l the theory developed in IITOl which show that, if the rolldown set obtained 
from a successful game of Betti poset pinball also satisfies poset-upper-triangularity conditions, then it 
yields a module basis in equivariant cohomology. In Sections S] and |5] we prove that the dimension pair 
algorithm produces a poset-upper-triangular module basis in a special class of regular nilpotent Hessenberg 
varieties which we call 334-type Hessenberg varieties. We close with some open questions in Section[6l 

Acknowledgements. We thank Erik Insko for explaining to us the correspondence between the elements 
of an affine paving of regular nilpotent Hessenberg varieties and certain Schubert polynomials which mo- 
tivates our dimension pair algorithm. We thank Barry Dewitt and Aba Mbirika for useful conversations 
and Rebecca Goldin for reviewing an initial draft of this manuscript and her many excellent suggestions 
for improving exposition. We are particularly indebted to Julianna Tymoczko for her ongoing support, for 
answering many questions, and for her suggestions on an earlier draft of this paper. 

2. Background 

We begin with necessary definitions and terminology for what follows. In Section 12.11 we recall the 
geometric objects and the group actions under consideration. In Section IZ2l we recall some combinatorial 
definitions associated to Young diagrams. We recall a bijection between Hessenberg fixed points and certain 
fillings of Young diagrams in Section l23l The discussion closely follows previous work (e.g. IIIOIIIII and 
also ITtI ) so we keep exposition brief. 

2.1. Hessenberg varieties, highest forms, and fixed points. By the flag variety we mean the homogeneous 
space GL{n, C)/B which is also identified with 

J£ags(C") {V, = ({0} C C C • ■ • K-i C K = C") | dimc{V,) = z}. 

A Hessenberg function is a fimction h : {1, 2, . . . , n} {1, 2, . . . , n} satisfying h{i) > i for all 1 < i < n 
and h(i + 1) > h{i) for all 1 < i < n. We frequently denote a Hessenberg function by listing its values in 
sequence, h = {h{l), h{2), . . . , h(n) — n). Let N : C" be a linear operator. The Hessenberg variety 

Hcss(iV, h) is defined as the following subvariety of J'£ags{C"'): 

(2.1) Hcss(Ar, h) {V, e JiagsiC") \ NV, C V^,) for alH = 1, . . . , n} C Jiags{U'). 

If N is nilpotent, we say Hcss(A^, h) is a nilpotent Hessenberg variety, and if N is the principal nilpotent 
operator (i.e. has one Jordan block with eigenvalue 0), then Hess(-/V, h) is called a regular nilpotent Hes- 
senberg variety. If N is nilpotent and h is the identity function h{i) = i for all 1 < i < n then Hcss(iV, h) is 
called a nilpotent Springer variety and often denoted Sjv. In this manuscript we study in some detail the 
regular nilpotent case, and as such sometimes notate Hess(A^, h) as Hcss(/i) when N is understood to be the 
standard principal nilpotent operator. 

Suppose given N a nilpotent matrix in standard Jordan canonical form. It turns out that for many of our 
statements below we must use a choice of conjugate of N which is in highest form [iTl Definition 4.2]. We 
recall the following. 

Definition 2.1. ( (l7l Definition 4.1 and Definition 4.2]) 
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• Let X be any m x n matrix . We call the entry Xi^ a pivot of X if Xn. is nonzero and if all entries 
below and to its left vanish, i.e., Xij = if j < k and Xjk = if j > i. Moreover, given i, define to 
be the row of X^^i if the entry is a pivot, and otherwise. 

• Let be an upper-triangular nilpotent n x n matrix. Then we say iV is in highest form if its pivots 
form a nondecreasing sequence, namely ri < r2 < • • • < r„ . 

We do not require the details of the theory of highest forms of linear operators; for the purposes of the 
present manuscript it suffices to remark firstly that when is the principal nilpotent matrix then is 
already in highest form, and secondly that any nilpotent matrix can be conjugated by an appropriate n x n 
permutation matrix a so that Nhj := aNa~^ is in highest form. However the following observation will be 
relevant in Section |2I51 

Remark 2.2. In this manuscript we always assume that our highest form N^f = aNa^^ has been chosen in ac- 
cordance to the recipe described by Tymoczko in ^T7\ Section 4]. Since the precise method of this construction is not 
relevant for the rest of the present manuscript we omit further explanation here. In the case when N is principal 
nilpotent we take Nhj = N since N is already in highest form and this is the form chosen by Tymoczko in iflTlf . A 
more detailed discussion of highest forms as it pertains to poset pinball theory is in iO. 

For details on the following facts we refer the reader to e.g. IITOllTTllTTl and references therein. Let 
be an n x n nilpotent matrix in Jordan canonical form and let a denote a permutation matrix such that 
Nhf := crNa^^ is in highest form. It is known and straightforward to show that the following subgroup 
of U (??, C) preserves Hcss(Af, h) for N as above and any Hessenberg fxmction h: 



(2.2) 








• 0' 
















. 








t 



t e C, = 1 ^ C T" C U{n,C). 



Here T" is the standard maximal torus of U (n, C) consisting of diagonal unitary matrices. 

This implies that the conjugate circle subgroup aS^a^^ preserves Hcss(A'^;i/, h). By abuse of notation we 
will denote both circle subgroups by S^, since it is clear by context which is meant. The S'^-fixed points of 
Hess(A^, h) and lless{Nfif, h) are isolated, and are a subset of the T"-fixed points of 3'£ags(C"). Since the 
set of T" -fixed points 3^£ags{C^)'^" may be identified with the Weyl group W = S'„, and since Hcss(A^, h)^^ 
(respectively Hess(A^;i/, h)^ ) is a subset of 3^£ags{C"')'^ , any Hessenberg fixed point may be thought of as 
a permutation w E Sn- 

2.2. Permissible fillings, dimension pairs, lists of top parts, and associated permutations. Recall that 
there is a bijective correspondence between the set of conjugacy classes of nilpotent n x n complex matrices 
A^ and Yoimg diagramqj with n boxes, given by associating to A^ the Young diagram A with row lengths the 
sizes of the Jordan blocks of A^ listed in weakly decreasing order. We will use this bijection to often treat such 
A^ and A as the same data; we sometimes denote by A at the Yoimg diagram given as above corresponding 
to a nilpotent A^. 

For more details on the following see IITSll . 

Definition 2.3. Let A be a Young diagram with n boxes. Let h : {1,2,. {l,2,...,n}bea Hessenberg 
function. A filling of A by the alphabet {1,2,..., n} is an injective placing of the integers {1,2,..., n} into 
the boxes of A. A filling of A is called a (h, A)-permissible filling if for every horizontal adjacency fc j in 
the filling we have k < h{j). 

Remark 2.4. In this manuscript the A and h will frequently be understood by context. When there is no danger of 
confusion we simply refer to permissible fillings. 



Example 2.5. Let n 

whereas 



2 3 4 1 5 



5. Suppose A = (5) and h 
is not, since 4 ^ ^(1). 



(3,3,4,5,5). Then 



2 


4 


3 


1 


5 



is a permissible filling, 



We denote a permissible filling of A by T, in analogy with standard notation for Young tableaux. Next 
we focus attention on certain pairs of entries in a permissible filling T. 



We use English notation for Young diagrams. 
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Definition 2.6. Let h : {1, 2, . . . , n} -> {1, 2, . . . , ri} be a Hessenberg function and A a Young diagram with 
n boxes. A pair (a, h) is a dimension pair of an [h, A) -permissible filling T of A if the following conditions 
hold: 

(1) b > a, 

(2) b is either 

• below a in the same column of a, or 

• anjrwhere in a column strictly to the left of the column of a, 
and 

(3) if there exists a box with filling c directly adjacent to the right of a, then b < h{c). 
For a dimension pair (a, b) of T, we will refer to b as the top part of the dimension pair. 



are 



(1, 2), (1, 3), and (1, 4). Note that (3, 4) is not a dimension pair because 1 is directly to the right of the 3 and 4 ^ h{l). 
Given a permissible filling T of A, we follow [13] and denote by DP^ the set of dimension pairs of T. 



Example 2.7. Let A, h be as in Example \T5\ The dimension pairs in the permissible filling 2 

(1. 2) , (1, 3), and (1, 4). Note that (3, 4) 

Given a permissible filling T of A, 
For each integer £ with 2 < £ < n, let 

(2.3) Xi:^\{{aJ)\{a,i)eDP'^}\ 

so xe is the number of times £ occurs as a top part in the set of dimension pairs of T. From the definitions it 
follows that < < £ — 1 for all 2 < £ < n,. We call the integral vector x = {x2,X3, . . . , a;„) the list of top 
parts of T. 

To each such x we associate a permutation in Sn as follows. As a preliminary step, for each £ with 
2<£<n define 

I Se^lSi^2 ■ • ■ Sl-xe ifXi>0 

II It Xe ^ 

where Si denotes the simple transposition + 1) in Sn and 1 denotes the identity permutation. Now 
define the association 



(2.4) X ^ w(x) := U2(x)m3(x) ■ • ■ u„(x) G Su- 

it is not difficult to see that (|2.4|l is a bijection between the set of integral vectors x G Z"^^ satisfying 
< xe < £ — 1 for all 2 < £ < n — 1 and the group Sn- In fact the word given by (|2.4) l is a reduced word 
decomposition of w(x) and the X( count the number of inversions in w(x) with £ as the higher integer. The 
following simple fact will be used later. 

Fact 2.8. Suppose x = (2:2, ... , x„), y = (1/2, • ■ • , Vn) £ ^>o^ ^'"^ ^ot/; /ists o/top parts. Suppose further that for all 
2 < £ < n, we have xe < ye- Then cj(x) < w(y) in Bruhat order. Thisfolloivs immediately from the definition (|2.4|| . 

Example 2.9. Continuing with Examples^^and\2.7\ for the permissible filling 2 | 4 3 15 the set DP'^ of 



top parts of dimension pairs is {2, 3, 4}, yielding the integer vector x = (1, 1, 1, 0). The associated permutation w(x) 
is then S1S2S3. 



Example 2.10. Let A, /i be as in Example \T5\ The filling 4 3 2 1 5 fs also permissible, with dimension pairs 



(1, 2), (1, 3), (1, 4), (2, 3). Hence x = (1, 2, 1, 0) and the associated permutation a;(x) is si(s2Si)s3. 

2.3. Bijection between fixed points and permissible fillings. For nilpotent Hessenberg varieties, the S^- 
fixed points Hcss(Af, h)^ are in bijective correspondence with the set of permissible fillings of the Yoimg 
diagram A = Aat, as we now describe. We will use this correspondence in the formulation of our dimension 
pair algorithm. 

Suppose A is a Young diagram with n boxes. We begin by defining a bijective correspondence between 
the set 3'i££{X) of all fillings (not necessarily permissible) of A with permutations in Sn- Given a filling, 
read the entries of the filling by reading along each column from the bottom to the top, starting with the 
leftmost column and proceeding to the rightmost column. The association 3^i££{X) o Sn is then given by 
interpreting the resulting word as the one-line notation of a permutation. For example the filling 





1 


2 


3 




4 


5 




T = 


6 
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has associated permutation 641523. It is easily seen that this is a bijective corresondence. Given a filling T 
of A we denote its associated permutation by 4>\{T). 

Remark 2.11. In the case when N is the principal nilpotent n x n matrix, the corresponding Young diagram A = 
A^v = {n) has only one row, so the above correspondence simply reads off the (one row of the) filling from left to 
right. In this case we abuse notation and denote 4)'^^ {w) by just w. For instance, the permissible filling of X = (5) in 
Example \2.10\ has associated permutation 43215. 

Now let 

(2.5) yFM{\ h) 

denote the set of (/i, A)-permissible fillings of A. Recall that elements in Hcss(A^, h)^ are viewed as permu- 
tations in Sn via the identification 3'£ags{C"')'^" = Sn- The next proposition follows from the definitions 
and some linear algebra. It is proven and discussed in more detail in Q, where the notation used is slightly 
different. 

Proposition 2.12. Fix n a positive integer. Let h : {1, 2, . . . , n} {1,2, ... ,n} be a Hessenberg function and A a 
Young diagram with n boxes. Suppose Nhf is a nilpotent operator in highest form as chosen in ilTTIf (cf. Remark \2^ 
with XNhf = ^- Hess(A^h./ , h) denote the associated nilpotent Hessenberg variety. Then the map from the S^-fixed 
points Hess(A^/i/, h)^^ to the set of permissible fillings 7FM{X, h) 

(2.6) w G Hess(/i)^' C 5„ ^ G TFiei{X,h) 
is well-defined and is a bijection. 

Remark 2.13. In the case when N is the principal nilpotent nxn matrix, X is the Young diagram with only one row. 
Thus the map \2.6\ above simplifies to w ^ w^^ where we abuse notation (cf. Remark ^.lli and denote ^^^(w^^) 
by w^^. 

3. The dimension pair algorithm for Betti poset pinball for nilpotent Hessenberg varieties 

In this section we first explain the dimension pair algorithm which associates to any Hessenberg fixed 
point a permutation in Sn- The name is due to the fact that the construction proceeds by computing dimen- 
sion pairs in appropriate permissible fillings. We then interpret this algorithm as a method for choosing 
roUdowns associated to the Hessenberg fixed points in a game of Betti poset pinball in the sense of [lOJ. 
The algorithm makes sense for any nilpotent Hessenberg variety, so it is defined in that generality in Sec- 
tion l3.1l However, our proof that the algorithm produces a successful outcome of Betti poset pinball in the 
sense of flOll is only for the special cases of regular nilpotent Hessenberg varieties and nilpotent Springer 
varieties. In Section |3^ we briefly recall the setup and necessary results of poset pinball which allow us to 
conclude that our poset pinball result yields an explicit module basis for equivariant cohomology. 

3.1. The dimension pair algorithm for nilpotent Hessenberg varieties. Let N^fhea nilpotent nxn matrix 
in highest form chosen as in Remark 12.21 and let A := ^N,,f Let h : {1, 2, . . . , n} {1, 2, . . . , n} be a 
Hessenberg function and lless{Nh / , h) the corresponding nilpotent Hessenberg variety. 

The definition of the dimension pair algorithm is pure combinatorics. It produces for each Hessenberg 

fixed point w G Hcss(A^h/, h)^ an element in Sn- Following terminology of poset pinball, we denote this 
function by 

roee:Ress{Nhf,hf' ^ Sn- 

Definition 3.1. ("The dimension pair algorithm") We define roii : liess{Nhf,h)^^ Sn as follows: 

(1) Let w G Hess(Af/,/, /i)'^ and let <l)'^^{w^^) be its corresponding permissible filling as defined in (12.611 . 

(2) Let DP'^>- ^ be the set of dimension pairs in the permissible filling (j}'^^{w~^). 

(3) For each i with 2 < £ < n, set 

X, :=|{(a,^) I (a,^) Gi^P*"("'")}| 

as in (|2.3l l and define x ;= {x2, ■ - ■ , Xn)- 

(4) Define roli{w) (aj(x))^^ where w(x) is the permutation associated to the integer vector x defined 
inllZl. 



POSET PINBALL, THE DIMENSION PAIR ALGORITHM, AND TYPE A REGULAR NILPOTENT HESSENBERG VARIETIES 



7 



Example 3.2. Let \,h he as in Example [231 The permutation w = 43215 S S'„ is in llcss{Nhf,h)^ 



be checked. The associated permissible filling is 4 3 2 1 5 . In Example 12.201 we saw that the associated 
permutation is si(s2Si)s3/ so we conclude roli{w) = sz{siS2)si. 

We next show that the roUdown function rait : Hcss(/i)'^^ — > 5„ defined by the dimension pair algorithm 
above satisfies the conditions to be a successful outcome of Betti poset pinball as in [10] in certain cases of 
nilpotent Hessenberg varieties. The statement of one of the conditions requires advance knowledge of the 
Betti numbers of nilpotent Hessenberg varieties, for which we recall the following result (reformulated in 
our language) from flTll . 

Theorem 3.3. ( ilTTl Theorem 1.1]) Let Nhf ■ C" — > C" be a nilpotent matrix in highest form chosen as in 
Remark \22\ and let \ Xnh}- h : {1,2, — > {1,2,. ..,n} be a Hessenberg fiinction and let Hess(iV/i/, /i) 
denote the corresponding nilpotent Hessenberg variety. There is a paving by (complex) affine cells ofllcsa{Nhf,h) 
such that: 

• the affine cells are in one-to-one correspondence with Hcss(A^/i/, h)^^ , and 

• the (complex) dimension of the affine cell Cw corresponding to a fixed point w G Hess(A^, h)^ is 

(3.1) dimc(C„) = li^P-^A |. 

In particular. Theorem 13.31 implies that the odd Betti numbers of Tless{Nhf,h) are 0, and the 2A;-th even 
Betti number is precisely the number of fixed points w in }icss{Nhf,h)^ suchthat IDP-^A (™ )|=fc. Given 
the regular nilpotent Hessenberg variety Hcss( A^;i f,h), denote by bk its 2fc-th Betti number, i.e. 

bk :=dimc H^''iRcss{Nhf,h)). 

We may now formulate the conditions that guarantee that ro££ : lless{NhfTh)^^ — > Sn is a successful 
outcome of Betti pinball. For more details we refer the reader to [10, Section 3]. It suffices to check the 
following: 

(1) roU : lless{Nhf,h)^^ Sn is injective, 

(2) for every w G Hess(iV/j/, /i)'^ , we have ro££{w) < w in Bruhat order, and 

(3) for every fc > 0, fc e Z, we have 



\roU{w) I w e Hcss(iVh/,/i)^' with €(ro«(u;)) = fcj 



where £{ro££(^w)) denotes the Bruhat length of ro££{w) £ Sn- 
We prove each claim in turn. For the first assertion we restrict to two special cases of Hessenberg vari- 
eties. 

Lemma 3.4. Suppose that llcss{Nhf, h) is either a regular nilpotent Hessenberg variety or a nilpotent Springer 
variety. Then the function ro££ : Hess(A^/i/, h)^^ — > Sn is injective. 

Proof. Since the association x = (2:2, a^s, ■ • ■ , a;„) n> aj(x) given in l|2.41 is a bijection it suffices to show that 
the map which sends a Hessenberg fixed point w £ Hcss(/i)'^^ to the list of top parts x of its associated 
permissible filling is injective. Mbirika shows that, in the cases of regular nilpotent Hessenberg varieties 
and nilpotent Springer varieties, there exists an inverse to this map (Mbirika works with monomials in 
n—1 variables constructed from the list of top parts, but this is equivalent data) Iil3i Section 3.2]. The result 
follows. □ 

Lemma 3.5. For every w G Hcss(/i)'^\ we have ro££{w) < w in Bruhat order. 

Proof. Since Bruhat order is preserved imder taking inverses, it suffices to prove that w(x) is Bruhat-less 
than w~^. For any permutation u G Sn, set 

y( := {(a, £) \ (a, £) is an inversion in u} 

and let y :— {1/2, 1/3, ... , ?/„)• Then the association (|2.4b applied to the vector y recovers the permutation u. 
By definition of (f>\ and the definition of dimension pairs, the set DP'^>- is always a subset of the set of 
inversions of the permutation w~^. From Fact 12.81 it follows that the permutation w(x) is Bruhat-less than 
as desired. □ 
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Lemma 3.6. Let N^j : C" C" he a nilpotent matrix in highest form chosen as in Remark \T2\ and let A :— ^n^/- 
Let h : {1,2,. ..,ri} — !> {1,2, be a Hessenber g function and B.ess{Nh,f,h) the associated nilpotent Hessenberg 
variety. For every k > 0,k e Z,zve have 

bk = |ro££(w) I w e Rcas{hf' with £{roe£{w)) = fcj , 

where £{ro££{w)) denotes the Bruhat length ofro££{w) G Sn- 

Proof. By construction, ro££(w) has a reduced word decomposition consisting of precisely |_DP'^a ('^ )| 
simple transpositions. Hence its Bruhat length is \DP'^>' '^^ •* I- By Theorem l3.3l bk is precisely the number 
of fixed points w with \DP'^>- ^ | = A: so the result follows. □ 



The following is immediate from the above lemmas and the definition of Betti pinball given in flOl 
Section 3]. 

Proposition 3.7. Suppose that Hcss(iV/i / , h) is either a regular nilpotent Hessenberg variety or a nilpotent Springer 
variety. Then the association w ro££{w) given by the dimension pair algorithm is a possible outcome of a suc- 
cessful game of Betti poset pinball played with ambient partially ordered set Sn equipped with Bruhat order, rank 
function p ~ £ : Sn ^ 1 given by Bruhat length, initial subset IIcss{h)^^ C Sn, and target Betti numbers 
bk :=dimci?2'=(Hess(/i);C). 



Remark 3.8. Lemmas [331 and [X6l hold for general nilpotent Nhj and Hessenberg functions h. Hence to prove that 
Proposition \3.7\ holds for more general cases of nilpotent Hessenberg varieties, it suffices to check that the injectivity 
assertion (1) above holds. We do not know counterexamples where the injectivity fails. It would be of interest to clarify 
the situation for more general N^f and h. 

3.2. Betti pinball, poset-upper-triangularity, and module bases. In the context of a GKM-compatible 
subspace of a GKM space fTOi Definition 4.5], it is explained in ITOl Section 4] that the outcome of a 
game of poset pinball may be interpreted as specifying a set of equivariant cohomology classes which, 
under additional conditions, yields a module basis for the equivariant cohomology of the GKM-compatible 
subspace. In this paper, the GKM space is the flag variety J'£ags{C") with the standard T"-action and 
the GKM-compatible subspace is Hcss(iV/i/, h) with the S'^-action specified above. Consider the (pt)- 
module basis for {3'£ags{U^ j) given by the equivariant Schubert classes {cTw}wes„ - The dimension pair 
algorithm then specifies the set 

{ProU(w) I w e Ilesa{Nhf,hf^} C Hgi(Iless{Nhf,h)) 

where for any u e Sn the class pu tt{(7u) is defined to be the image of (t„ imder the natural projection 
map 

TT : i7^„(J^ags(C")) ^ H*i{}iess{Nhf,h)) 

induced by the inclusion of groups S^ ^ T" and the S'^-equivariant inclusion of spaces Hcss(iV/jj^, h) ^ 
3^£ags{C^). We refer to the images p„ as Hessenberg Schubert classes. 

Following the methods of IITOll we view _ff^„ {y£ags{'C'"')) and Hgi (Hcss(Af^j, h)) as subrings of 

77^„((mgs(C"))^") ^ i?|.„(pt) respectively iJ^i ((Hcss(iV,,/, /i))'^') = i?si(pt)- 

We denote by cF.u]{w'),ProU(w) [w') the value of the w'-th coordinate in the direct sums above, for 
or w,w' G Hcss(iV/i/, h)^^ respectively. If 

(3.2) Proee{w}{w) ^ 0, and Proee{w}{w') = if w ^ w' 

for all w,w' G 'Hcss{Nhf,h)^^ then the set {proU(w) I w G }icss{Nhf,h)^'^} in Hgi{Hess{Nhf,h)) is called 
poset-upper-triangular (with respect to the partial order on llcss{Nhf , h)^ C Sn induced from Bruhat 
order) [10, Definition 2.3]. Finally, recall that the cohomology degree of an equivariant Schubert class a^j 
(and hence also the corresponding Hessenberg Schubert class pu,) is 2 • £{w). 

The following is immediate from [10. Proposition 4.14] and the above discussion. 
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Proposition 3.9. Let Hgss(A^^^, h) he either a regular nilpotent Hessenberg variety or a nilpotent Springer variety. 
Let ro££ : liess{Nhf, h)^^ — > Sn be the dimension pair algorithm defined above. Suppose (|3.2t holds for all w e 
Hess(A^/i/, /i)"^ . Then the Hessenberg Schubert classes {proU{w) \ w S 'Kess{Nhf,hY } form a Hgi {pt)-module 
basis for the S^-equivariant cohomology ring Hgi (Hess(A^/i/, h)). 

Therefore, in order to prove that the Hessenberg Schubert classes above form a module basis as desired, 
it suffices to show that they satisfy the upper-triangularity conditions (|3.2| | for all w' S Hcss(A'^;i/, h)^ . 
The proof of this assertion, for a special class of regular nilpotent Hessenberg varieties closely related to 
Peterson varieties, is the content of Sections |4] and |5l 

We close the section with a brief discussion of matchings. Following flOl Section 4.3], define 

^(^SHcss{N^fji)M dinic(a«) 

to be the (complex) dimension of the affine cell containing the fixed point w in Tymoczko's paving by 
affines of Hess(iV/i/, h) in Theorem l3.3l Then from the discussion above we know 

dcgHcss(7V.„h)(«^) = = i{roie{w)), 

and since the cohomology degree of Proee{w) is 2 • £{roU{'w)), we see that the association w roi£{w) from 
Kess{Nhf, h)^'^ Sn is also a matching in the sense of IITOl with respect to degncssiNhf ,h) ^^id rank function 
p on Sn given by Bruhat length. Thus the fact that the {prott{w) I w & Hess(A^^/, h)^^} form a module basis 
can also be deduced from [lOl Theorem 4.18]. 

4. POSET-UPPER-TRIANGULARITY OF ROLLDOWN CLASSES FOR 334-TYPE HESSENBERG VARIETIES 

In this section and in Section |5] we analyze in detail the dimension pair algorithm in the case of a Hes- 
senberg variety which is closely related to the Peterson variety, and in particular prove that the algorithm 
produces a poset-upper-triangular module basis for its 5^-equivariant cohomology ring. Here and below 
the nilpotent operator N under consideration is always the principal nilpotent, so we omit the from the 
notation and write Hess(/i). Similarly the corresponding Young diagram is always A = (n) so we omit the 
A from notation and write 'PFM{h) instead of 'PFM{\ h). 

We fix for this discussion the Hessenberg function given by 

(4.1) = /7,(2) = 3, = i + 1 for 3 < i < n - 1, and = n. 

The only difference between this function h and the Hessenberg function for the Peterson variety studied 
in lim is that the value of h{\) is 3 instead of 2. In this sense this h is "close" to the Peterson case. Thus it 
is natural that much of our analysis follows that for Peterson varieties in fTT], although it is still necessary 
to introduce new ideas and terminology to handle the Hessenberg fixed points in Hcss(/i)'^ which do not 
arise in the Peterson case. 

The Hessenberg function h in ll4.lt is trivial if 7i = 3 since in that case h{l) = h{2) = h{3) = 3 which 
implies that the corresponding Hessenberv variety Hess(/i) is equal to the full flag variety J'iags{C^). Hence 
we assume n > 4 throughout. Under this assumption and following the notation introduced in Section |2| 
the Hessenberg function is of the form h = (3, 3, 4, ■ • ■ ). As such, for the purposes of this manuscript, we 
refer to this family of regular nilpotent Hessenberg varieties as 334-type Hessenberg varieties. 

Our main result is the following theorem. 

Theorem 4.1. Lef n > 4 and let Hess(/i) be the 334-type Hessenberg variety in Jiags{C"-). Let ro££ : Hess(/i)'^^ — > 
Sn be the dimension pair algorithm defined in Section\3\ Then 

(4.2) ProU{w)iw) 0, and ProU{w)iw') ^ if w ^ w' 

for all w,w' G Hess(/i)'^\ In particular the Hessenberg Schubert classes {ProU{w) 1 w e lless{h)^^} form a 
Hgi {pt)-module basis for the S^-equivariant cohomology ring Hgi (Hcss(/i)). 

For ease of exposition, and because the arguments required are of a somewhat different nature, we prove 
Theorem l4.1l by proving the two assertions in (|4.2b separately, as follows. 
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Proposition 4.2. Let n, h, Hcss(/i) and ro££ be as above. Then 

(4.3) ProU{w)iw)^0 

for all w G Hcss(/i)'^\ 

Proposition 4.3. Let n, /i, Hcss(/i) and ro££ be as above. Then 

(4.4) ProU{w) {w') =Q\{w ^w' 
for all w, w' e Hess(/i)'^^ . 

The proof of Proposition 14.21 is the content of Section |5] The main result of the present section is the 
upper-triangularity property asserted in Proposition l4.3l Its proof requires a number of preliminary results. 
We first begin by reformulating the problem in terms of Bruhat relations among the fixed points. 

Lemma 4.4. Let n, h, Hess(/i) and ro££ he as above. If for all w, w' £ Hess(/i)'^^ we have 

(4.5) ro££{w) <w' <w' 

in Bruhat order, then the Hessenberg Schubert classes {ProU(w) I w e Hess(/i)'^^} satisfy (|4.4b . 

Proof. Recall that the equivariant Schubert classes are poset-upper-triangular with respect to Bruhat order 
on Sn- In particular, for all w, w' G Sn we have cr„, {w') = if w' ^ w. Since the Hessenberg Schubert classes 
are images of the Schubert classes and the diagram 

(4.6) i/f.(J^ags(C"))C > H^„{{J£ags{C"))^" ) - e^evv^T^Pt) 



commutes, it follows that if for all w, w' G Hess(/i)'^\ we have 

(4.7) ro££{w) < w' ^ w < w' 

in Bruhat order then l|4.4|l follows. □ 

The rest of this section is devoted to the proof of l|4.5|l . which by Lemma l44l then proves Proposition |43l 

4.1. Fixed points and associated subsets for the 334-type Hessenberg variety. In this section we enumer- 
ate the fixed points in the 334-type Hessenberg variety and also associate to each fixed point in Hcss(/i)'^ 
a subset of {1,2, ... ,n — 1}. As we show below, the set of fixed points in the Peterson variety is a subset 
of the fixed points of the 334-type Hessenberg variety, so the main task is to describe the new fixed points 
which arise in the 334-type case. We begin with a general observation. 

Lemma 4.5. Let n G N and let h, h' : {1,2,..., 71} -> {1, 2, . . . , n} be two Hessenberg functions. lfh{i) > h'{i) 
for all 1,1 < i < n, then 

Ress{h') C Hess(/i). 

The inclusion Hcss(/i') ^ Hess(/i) is -equivariant and in particular Hcss(/i')'^^ C Hcss(/i)'^^ and 7Fi££{h') C 
yFMQi). 

Proof. Let V, ~ {Vi) denote an element in J'£ags{C"). By definition the regular nilpotent Hessenberg variety 
Hess(/i') associated to h' is 

(4.8) Hcss(/i') := {V. G J£ags{U') \ NV^ C V,,,(,;), for all 1 < i < n} 

where N is the principal nilpotent operator. Since Vi C for all 1 < i < 77, — 1 by definition of flags 
and y„ = C" for all flags, if h'{i) < h{i) for all i then NV.^ C V^,(,) automatically implies NVi C Vh{^. We 
conclude Hess(/i') C Hess(/i). The S^-equivariance of the inclusion Hess(/i') Hess(/7) follows from the 
definition of the 5^-action of l|2.2b . □ 
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Applying Lemma |45l to the Hessenberg fimction 
(4.9) h'{i) = i + 1 for 1 < i < n - 1, h'{n) = n 

corresponding to the Peterson variety Hcss(/i') and h the 334-type Hessenberg function ll4.Hl , we conclude 
that all fixed points in Hess(/i')'^ also arise as fixed points in Hcss(/i)'^ . We refer to the elements of 
Hess(/i')'^ (viewed as elements of Hcss(/i)'^ ) as Peterson-type fixed points. It therefore remains to describe 
Hess(/i)'^ \ Hess(/i')'^ . It turns out to be convenient to do this by first describing 'J'FM{h) \ 7FiU{h'). 

We first introduce some terminology. Given a permutation w = (w(l) w{2) ■ ■ ■ w{n)) in one-line notation 
and some i, £, we say that the entries {w{i),w{i + 1), . . . , w{i + form a decreasing staircase, or simply a 
staircase, if w{j + 1) = w{j) — 1 for all i < j < i + £. For example for w = 4327516, the segment 432 is a 
staircase, but 751, though the entries decrease, is not. We will say that a consecutive series of staircases is 
an increasing sequence of staircases (or simply increasing staircases) if each entry in a given staircase is 
smaller than any entry in any following staircase (reading from left to right). For instance, w = 654987321 
is a sequence of staircases 654, 987, and 321, but is not an increasing sequence of staircases since the entries 
4, 5, 6 are not smaller than the entries in the later staircase 321. However, w ~ 321654987 is an increasing 
sequence of (three) staircases 321, 654, and 987. 

It is shown in IITTl that the S'^-fixed points of the Peterson variety Hess(/i') consist precisely of those 
permutations w G Sn such that the one-line notation of w is an increasing sequence of staircases. Since such 
w are equal to their own inverses, the permissible fillings yFM{h') corresponding to Hess(/i') are precisely 
those which are increasing sequences of staircases (cf. Remark 12.131 1. We now describe the permissible 
fillings 7FM{h) which are not Peterson-type fillings. We use the language of /i-tableau trees introduced by 
Mbirika; see IITSl Section 3.1] for definitions. Recall from Remark 12.111 that we identify permissible fillings 
with permutations in 5„ via one-line notation. 

Lemma 4.6. Let n > 4 and let Hess(/i) he the SSi-type Hessenberg variety in 3^£ags{C"). Let w S ?FiU{h) be a 
permissible filling for Hess(/i) which is not of Peterson type, i.e., w G 7Fiii{h) \ 7FM{h'). Then precisely one of 
the following hold: 

• The one-line notation ofw is of the form 

w' 3 1 2 w" 

where w' is a (possibly empty) staircase such that w' 3 is also a staircase, and w" is an increasing sequence of 
staircases. We refer to these as 312-type permissible fillings. 

• The one-line notation ofw is of the form 

2 3 1 w" 

where w' is a (possibly empty) staircase such that w' 3 is also a staircase, and w" is an increasing sequence of 
staircases. We refer to these as 231-type permissible fillings. 
Moreover, any filling satisfying either of the above conditions appears in 'PFU£{h) \ 'PFU£{h'). 

Proof of Lemma 1416] For any Hessenberg function h : {1, 2, . . . , n} {1,2,..., n}, Mbirika shows in lT3l 
Section 3.2] that the Level n fillings in an /i-tableau tree are precisely the permissible fillings with respect to 
h. For the Peterson Hessenberg fimction in I l4.9l l Mbirika's corresponding /i-tableau tree has the property 
that for every k with 1 < fc < n — 1 and every vertex at Level k, there are precisely 2 edges going down from 
that vertex to a Level fc + 1 vertex. (This is because the corresponding degree tuple /3 (131 Definition 3.1.1] has 
Pi = 2 for all 1 < i < ?i — 1.) In the case of the 334-type Hessenberg function, by definition the /i-tableau tree 
also has precisely 2 edges going down from every vertex at Level fc for all fc 7^ 2, 1 < fc < n — 1 . However, 
at Level 2, each vertex has not 2 but 3 edges pointing down to a vertex at Level 3. 

From IIT3I Section 3] (cf. in particular lT3l Definition 3.1.9]) it can be seen that for the case of the Peterson 
Hessenberg function, the corresponding /i-tableau tree at Level 2 has vertices • 2 1 • and 1 • 2 •, whereas 
for the 334-type Hessenberg function, the Level 2 vertices have the form • 2 • 1 • and • 1 • 2 •. Here the 
bullets indicate the locations of the /i-permissible positions available for the placement of the next index 3, 
in the sense of [131 Section 3] (cf. in particular [131 Lemma 3.1.8]). In particular, since we saw above that 
the edges going down from Level 3 onwards are identical in both the Peterson and 334-type Hessenberg 
case, it follows that the branches of the tree emanating downwards from the two Level 3 vertices 3 2 1 •, 
2 1 3 • (coming from • 2 • 1 •) and the two vertices 1»32», 12»3» (coming from • 1 • 2 •) are identical to the 
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corresponding branches in the /i-tableau tree for the Peterson Hessenberg function. Hence all permissible 
fillings at the final Level n of these branches are of Peterson type. In contrast, the branches emanating 
from 2 • 3 1 • and • 3 1 2 • do not appear in the Peterson /i-tableau tree, and none of the fillings appearing at 
Level n in these branches can be Peterson permissible fillings since a 3 appears directly before a 1 . Hence 
it is precisely these branches which account for the permissible fillings which are not of Peterson type. 
As noted above, the rest of the branch only has 2 edges going down from each vertex with /i-permissible 
positions determined exactly as in the Peterson case. In particular, except for the exceptional 3 appearing 
directly to the left of a 1, the fillings must consist of decreasing staircases and all possible arrangements of 
decreasing staircases do appear. The result follows. 

□ 

Example 4.7. Suppose n ~ 8. Then w ~ 54312876 is an example of a 312-type permissible filling where w' = 54 
and w" = 876. An example of a 231-type permissible filling is w = 25431876 where w' = 54 and w" = 876. Neither 
of these are permissible with respect to the Peterson Hessenberg function h' since a 3 appears directly to the left of a 
1. Nevertheless, both of these fillings are closely related to the Peterson-type permissible filling w = 54321876; this 
relationship is closely analyzed and used below. 

We now give explicit descriptions of the corresponding non-Peterson-type elements in Hess(/i)'^ , ob- 
tained by taking inverses of the permissible fillings described in Lemma [4.6l 

Definition 4.8. Let w E Hcss(/i)'^^ . We say w is a 312-type (respectively 231-type) fixed point if its inverse 
w^^ is a permissible filling of 312-type (respectively 231-type). 

As observed above, since Peterson-type permissible fillings are equal to their own inverses, in that case 
there is no distinction between the fillings and their associated fixed points. For the 312 and 231-types, 
however, this is not the case. We record the following. The proof is a straightforward computation and is 
left to the reader. 

Lemma 4.9. Let w be a 312-type (respectively 231-type) permissible filling. Let a2 be the integer such that 02 + 1 
is the first entry (respectively second entry) in the one-line notation of w. Let be the corresponding 312-type 
(respectively 231 type) fixed point. Then: 

• the one-line notation ofw~^ is the same as that of w for all £-th entries with £ > a2 + 1, 

• ifw is 312-type, then the first 02 + 1 entries of the one-line notation ofw^^ are 

(4.10) a2 a2 + 1 a2 - 1 a2 - 2 • • • 2 1 

• ifw is 231-type, then the first 02 + 1 entries of the one-line notation ofw~^ are 

(4.11) a2 + 1 1 a2 a2 - 1 ■••32 

In the case of the Peterson variety, there is a convenient bijective correspondence between the set of S'^- 
fixed points of the Peterson variety and subsets yi of {1, 2, . . . , n — 1} given as follows llllt Section 2.3]. Let 
w be a Peterson-t5rpe fixed point. Then the corresponding subset is 

(4.12) A := {i -.1 <i <n-l and w{i) = w(i + 1) + 1} C {1, 2, . . . , n - 1}. 

In the case of the 334-type Hessenber variety, it is also useful to assign a subset of{l,2,...,n — l}to each 
fixed point as follows. 

Definition 4.10. Let w e Hess(/i)'^^ . The associated subset of {1, 2, . . . , n} corresponding to w, notated 
A{w), is defined as follows: 

• Suppose w is of Peterson type. Then is defined to be the set A in l|4.121 . 

• Suppose w is 312-type. Consider the permutation w' := wsi (i.e. swap the 02 and the a2 + 1 in the 
one-line notation l|4.10ll ). This is a fixed point of Peterson t5'pe. Define A{'w) := A{'w'). 

• Suppose w is 231-type. Consider the permutation 

w' = WS2S^ ■ ■■ Sa2 

(i.e. move the 1 to the right of the 2 in the one-line notation l l4.1Hl ). This is a fixed point of Peterson 
type. Define A{w) := A{w'). 

Example 4.11. Suppose n = 8. 
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• Suppose w is the Peterson-type fixed point w = 54321876. Then A{w) = {1, 2, 3, 4} U {6, 7}. This agrees 
with the association w A{w) used in iflTlf . 

• Suppose w is the 312-type fixed point w = 34217658 (corresponding to the 312-type permissible filling 
43127658). Then w' = wsi ^ 43217658 and A{w) A{w') = {1, 2, 3} U {5, 6}. 

• Suppose w is the 231-type fixed point w = 51432768 (corresponding to the 231-type permissible filling 
25431768). Then w' = 54321768 and A{w) := A{w') = {1, 2, 3, 4} U {6}. 

Remark 4.12. The three fixed points w = 54321876, w = 45321876, and w ~ 51432876, zvhich are respectively of 
Peterson type, 312 type, and 231-type, all have the same associated subset A{w) ~ {1,2, 3, 4} U {6, 7}. 

It is useful to observe that the 312-type and 231-type fixed points have associated subsets that always 
contain 1 and 2. 

Lemma 4.13. Let w be a 334-type Hessenberg fixed point. Suppose further that w is not of Peterson type. Then 
{1,2} C A{w). 

Proof. From the explicit descriptions of the one-lrne notation of the 312 type (respectively 231-type) fixed 
points given above, we know that the initial segment a2 a2 + 1 ■ • • 21 (respectively 02 + 1 1 a2 • • ■ 3 2) rn the 
one-line notation is such that 02 > 2. From Definition 14 . 1 01 it follows that the first decreasing staircase of the 
associated Peterson-type fixed point wsi (respectively ws2S^ ■ • ■ 502) is of length at least 3. In particular, the 
first staircase starts with an integer k which is > 3. The result follows. □ 

As noted in Remark|4A2l the association w ^ A{w) given in Definition 14. 1 01 is not one-to-one and hence 
in particular not a bijective correspondence. This makes our analysis more complicated than in fTTll , but the 
notion is still useful for our arguments below. 

4.2. Reduced word decompositions for 334-type fixed points and rolldowns. In this section we fix par- 
ticular choices of reduced word decompositions for the fixed points in Hess(/i)"^^ which we use in our 
arguments below. We also compute, and fix choices of reduced words for, the rolldowns roM{w) of the 
fixed points. 

The association w i—> A{w) of the previous section allows us to describe these reduced word decomposi- 
tions in relation to that of the Peterson-type fixed points. Let a be a positive integer and k a non-negative 
integer. Recall that a reduced word decomposition of the maximal element (the full inversion) in the sub- 
group S{a.a+i.,....a+k+i} ^ Sn IS giveu by 

(4.13) SaiSa+lSa){Sa+2Sa+lSa) ' ' ' {Sa+kSa+k-1 ' ' ' Sa+lSa)- 

For the purposes of this manuscript, we call this the standard reduced word (decomposition) for the maximal 
element. (This is different from the choice of reduced word decomposition used in flli Section 2.3].) We 
denote a consecutive set of integers {a,a + 1, . . . ,a + k} for a positive and k a non-negative integer by 
[a, a + k]. We say that [a, a + A:] is a maximal consecutive substring of A if [a,a + k] C A and neither a — 1 
nor a + k + 1 are in A. It is straightforward that any subset yiof{l,2,...,n — 1} uniquely decomposes into 
a disjoint union of maximal consecutive substrings 

(4.14) A = [ai,a2] U [03, 04] U • • • U [a„i_i, a™]. 

For instance, for yi = {1, 2, 3, 5, 6, 9, 10, 11}, the decomposition is yi = [1, 3] U [5, 6] U [9, 11]. For any [a, b], 
denote by w^^^b] the full inversion in the subgroup S[a,b+i] ■ Then it follows from Defrnition l4.10l (see also fill 
Section 2.3]) that the Peterson-type fixed point associated to A, which we denote by wji, is the product 

We fix a choice of reduced word decomposition of wa given by taking the product of the standard reduced 
words l l4.13ll for each of the full inversions wiaj .aj+i] appearing in l l4.15ll . For the purposes of this manuscript 
we call this the standard reduced word decomposition of a Peterson-type fixed point wa- 

Example 4.14. Let n = 7 and let w = 4321765 be a Peterson-type fixed point. Then the two decreasing staircases 
are 4321 and 765, the associated subset A{'w) is {1, 2, 3} U {5, 6} with maximal consecutive strings [1,3] := {1,2,3} 
and [5, 6] := {5, 6}. The standard reduced word decomposition ofw is 

(4.16) U'{l,2,3}U{5,6} = 1i'[l,3]'i«[5,6] = Si (S2S1 ) (s3S2Si)s5 (sgSg) . 

We now fix a reduced word decomposition of the non-Peterson-type fixed points. 
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Lemma 4.15. Let w G Hcss(/i)'^^ be a fixed point which is not of Peterson type and let A{w) = [ai, 02] U [03, 04] U 
• ■ • U be the associated subset with its decomposition into maximal consecutive substrings. 

• Ifw is 312-type then a reduced word decomposition for w is given by 

(4.17) si(s2Si)-..(s 
and 

• ifw is 231-type then a reduced word decomposition for w is given by 

(4.18) 82(8332) ■ ■ ■ {Sa2-lSa2~2 ' ' ' S3S2 ) (Sas ^aa -1 ' ' " ^2 Sl)W[a3,Q4] ' ' ' W[a„_i,a™] 

where the W[ai,,ae+i] in the above expressions are assumed to be given the reduced word decomposition described 
in diia . 

Proof. For the first assertion, observe that the explicit description of the one-line notation 312-type fixed 
points in | |4.10|| implies that w has precisely 1 fewer inversion than Wji(w)- An explicit computation shows 
that the given word l|4.17t is equal to w, so it is a word decomposition of w with exactly as many simple 
transpositions as the Bruhat length of w. In particular it must be reduced. A similar argument proves the 
second assertion. □ 

Example 4.16. Suppose n = 7. Suppose w = 3421765 is a 312-type fixed point. Then the reduced word decomposi- 
tion ofw given in Lemma \4?T5\ is 

W = S1(S2S1)(S3S2)S5(S6S5). 

Similarly suppose w = 4132765 is a 231-type fixed point. Then the reduced word decomposition of w given in 
Lemma \4.15\ is 

W = S2(s3S2Si)s5(s6S5). 

Henceforth we always use the reduced words given above. 

Next we explicitly describe the rolldowns roU{w) associated to each w in Hcss(/i)"^ by the dimension 
pair algorithm. We begin with the Peterson-type fixed points. It turns out there are two important subcases 
of Peterson-type fixed points. 

Definition 4.17. We say that a Peterson-type fixed point w contains the string 321 (or simply contains 321) 
if, in the one-line notation of w, the string 321 appears (equivalently, if {1, 2} C A{w)). We say w does not 
contain the string 321 (or simply does not contain 321) otherwise. 

Remark 4.18. Note that Definition \4.17\ is different from the standard notion of pattern-containing or pattern- 
avoiding permutations since here we require the one-line notation ofw to contain the string 321 exactly. 

Given a subset A = {ji < j2 < ■ ■ ■ < jk} ^ {1, 2, . . . , n — 1} and corresponding Peterson- type fixed point 
wa, we call the permutation 

(4.19) 8j^Sji,_^ ■ ■ ■ 8j^Sj-^ S Sn 

the Peterson case rolldown of w^ . Note that the word (|4.19l l is in fact a reduced word decomposition of 
this permutation; we always use this choice of reduced word. The terminology is motivated by the fact 
that ll4.19l l is the (inverse of the) permutation given in |11. Definition 4.1]. (The fact that it is the inverse of 
the permutation used in fTl] does not affect the theory very much, as is explained in [TTl Proposition 5.16].) 

Lemma 4.19. Let n> 4 and Hcss(/i) the 334-type Hessenberg variety in JiagslC"). Let wbea Peterson-type fixed 
point and let A{w) = {ji < j2 < ■ ■ ■ < jk} be its associated subset. 

• Suppose w does not contain 321. Then roi£{w) is the Peterson case rolldown ofwA{w)- 

• Suppose w does contain 321, i.e., A{w) = {ji < j 2 < • • ■ < jk} for k > 2 and ji = 1 and j2 = 2. Then 

roU{w) is 

(4.20) roU{w) = 8j^Sj^_, ■ ■ ■ 8j.,8iS2Si. 

In particular, if a Peterson-type fixed point w contains 321, then its rolldown ro££{w) is Bruhat-greater, and has 
Bruhat length 1 greater, than the Peterson case rolldown ofw. 
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Proof. If w contains a 321, then by Definition 12. 6[ the pairs (1, 3), (2, 3) and (1, 2) are all dimension pairs 
in w. Hence 3 appears precisely twice as a top part of a dimension pair and 2 appears precisely once. 
Thus by construction the dimension pair algorithm the permutation cl)(x) begins with the word si(s2Si)- 
With respect to all other indices j G A(w), the 334-type Hessenberg function is identical to the Peterson 
Hessenberg function and hence for each such j, the index j + 1 appears precisely once as a top part of 
a dimension pair of w and thus contributes precisely one Sj to w(x). Taking the inverse yields (|4.20t as 
desired. 

If w does not contain 321, then 3 appears at most once as the top part of a dimension pair in w, and again 
for all other indices the computations are identical to the Peterson case as above. Hence ro£i{w) is identical 
to the Peterson case rolldown. This completes the proof. □ 

Next, we give an explicit description, along with a choice of reduced word decomposition, of the roll- 
downs corresponding to the non-Peterson-type fixed points. 

Lemma 4.20. Let w G Hess(/i)'^^ and suppose that w is not of Peterson type. Let A{w) = {ji = 1 < ^2 = 2 < 
is < ■ • • < jkjfor some k>2. 

(1) Ifw is of3l2-type, then the dimension pair algorithm associates to w the permutation 

(4.21) roU{w) = Sj^Sj^_^ ■ ■ ■ s^,Sj^siS2. 

(2) Ifw is 22>l-type, then the dimension pair algorithm associates to w the permutation 

(4.22) roU{w) = Sj^Sj,_, ■ ■ ■ Sj^Sj^S2Si. 

Proof. Suppose w is a 312-type fixed point so is a 312-type permissible filling. By definition of 

dimension pairs, 2 does not appear as the top part of any dimension pair (since it appears to the right of 
a 1). Also by definition, 3 appears as a top part of the two dimension pairs (1, 3) and (2, 3). The form of 
the 312-type permissible fillings described in Lemma |46l and the definition of A{w) imply that the other 
dimension pairs are precisely the pairs (j, + 1) for j G A{w) (for j ^ 1, 2), from which it follows that 
a;(x) = S2SiSj^Sj^ ■ ■ ■ Sj^ -^Sj^ . Taking inverses yields ||4.21| |. The proof of the second assertion is similar. □ 

Example 4.21. • Suppose w = 54321876. This is of Peterson type. Thenroi£{w) ~ (w(x))^^ — s-js^s^szS\S2S\. 

• Suppose w = 45321876. This is 312-type. Then ro£i{w) = (a;(x))^^ = sjseS4^S3SiS2. 

• Suppose w = 51432876. This is 231-type. Then ro£i{w) = (w(x))^^ = sjseS4^S3S2Si. 

We conclude the section with a computation of the one-line notation of the rolldowns for different types; 
we leave proofs to the reader. 

Lemma 4.22. Let wbea 33A-type Hessenberg fixed point and let A{w) = [oi , 02] U • • • U [a,„_i , a„i] be its associated 
subset with its decomposition into maximal consecutive substrings. Suppose w is of Peterson type that contains 321, 
312-type, or of 231 type. Then ai = 1, 02 > 2 and the first 02 + 1 entires of the one-line notation ofroii{w) is 

(4.23) 02 + 12134 •■• 02 
for w of Peterson type that contains 321, 

(4.24) 2 02 + 1 1 3 4 • • • 02 
for w 312-type, and 

(4.25) 02 + 1 1 2 3 • ■ • a2 
for w 231-type. 

4.3. Bruhat order relations. In this section we analyze the properties of the association w A{w) with 
respect to comparisons in Bruhat order. 

The first two lemmas are straightforward and proofs left to the reader. 

Lemma 4.23. Let yic{l,2,...,n — 1} and let wa be the Peterson-type filling associated to A. Then wji is maximal 
in the subgroup Sa of Sn generated by the simple transpositions {sijigyi- in particular, wa is Bruhat-bigger than 
any permutation w & Sa- 

Lemma 4.24. Let w G Hess(/i)'^\ Suppose w is not of Peterson type. Then w is Bruhat-less than the Peterson type 
fixed point wa(w) corresponding to A{w). 
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We also observe that a Bruhat relation w < w' implies a containment relation of the associated subsets. 

Lemma 4.25. Let w,w' £ Hess(/i)'^^ and let A{w),A{w') be the respective associated subsets. Let Si be a simple 
transposition. Then: 

(1) Si < w if and only ifi G A{'w), 

(2) Si < ro££{w) if and only ifi £ A{w), 

(3) ifw < w' or roU{w) < w' then A{w) C A{w'). 

Proof. Bruhat order is independent of choice of reduced word decomposition for w. Therefore a simple 
transposition Si is less than w in Bruhat order if and only if s; appears in a (and hence any) reduced word 
decomposition of w. In particular, to prove the first claim it suffices to observe that by the definitions of 
A{w), the index i appears in A{w) precisely when Si appears in the choice of reduced word for w given 
above. A similar argument using the explicit reduced words given for ro(.£{w) in Lemmas 14.191 and 14.201 
proves the second claim. The last claim follows from the first two. □ 

We have just seen that w < w' implies A{w) C A{w'). In the case of the Peterson variety Hcss(/i') 
these Bruhat relations are precisely encoded by the partial ordering given by containment of the A{w); 
specifically, by Lemma |4.23[ wji < if and only if yi C 2. In our 334-type Hessenberg case this is no 
longer true, although the sets A{w) do still encode the Bruhat data. The precise statements occupy the next 
several lemmas. 

We take a moment to recall the tableau criterion for determining Bruhat order in the Weyl group 5„ (see 
e.g. |3l) which will be useful in the discussion below. For w e Sn, denote by Dr{w) the descent set of w, 
namely, 

Dr{w) := {i I w{i) > w{i + < i < n - 1}. 
For example, for w = 368475912 the descent set is Dr{w) = {3, 5, 7}. 

Theorem 4.26. ("The tableau criterion" Theorem 2.6.3]) For w,v £ Sn, let Wi^u be the i-th element in the 
increasing rearrangement o/w(l), w(2), . . . , w{k), and similarly for Vi,k. Then w < v in Bruhat order if and only if 

(4.26) Wi^k < Vi,k for all k e Dii{w) and 1 <i <k. 

For example, suppose w = 368475912 and v = 694287531. Since Dr{w) = {3, 5, 7}, we examine the three 
increasing rearrangements of initial segments of w and v of lengths 3,5, and 7 respectively, which we may 
organize into Young tableaux: 
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Comparing corresponding entries, there are two violations of the tableau condition of the proposition (3 > 
2) in the upper-left corner, so we conclude that w </i v. 
Now we observe that some Bruhat relations never arise. 

Lemma 4.27. Let w, w' E Hcss(/i)'^\ Let A{w) = [ai, 02] U • • ■ U [am-i,ani] be the associated subset of w with its 
decomposition into maximal consecutive substrings. Suppose one of the following conditions hold: 

(1) w' is of Peterson type that does not contain 321 while w is not, 

(2) w' is 251-type while w is either of Peterson type that contains 321 or is 312-type. 
Then w -ft w' and ro££{w) ^ w' . 

Proof. If w' is of Peterson type that does not contain 321, then {1,2} % A{w') by definition of the associated 
subsets. All other types (Peterson type that contains 321, or 312-type, or 231-type) have associated subsets 
containing {1, 2} by Lemma [4. 131 and by definition of A{'w). The claim (1) now follows from Lemma [4.251 

Next suppose w' is 231-type and w is of Peterson type that contains 321. Then the first two entries of 
the one-line notation of w must be both strictly greater than 1, and 2 e Dnfw). Similarly if w is a 312-type 
fixed point then 02 > 2. From | |4.10|| it follows that the first two entries in the one-line notation of w are 
also strictly greater than 1, and 2 G Dj^{w). On the other hand, the one-line notation for a 231-type fixed 
point in l|4.11t has a 1 in the second entry. By the tableau criterion, if w < w' then since 2 G D}i{w) in both 
cases under consideration, we must have that one of the first two entries of w is equal to 1, but we have 
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just seen that is impossible. Hence w it w' . The assertion that ro££{w) w' follows by a similar argument 
using l l423ll , ll424l l, and □ 

For the next lemma and below, we say two fixed points are of the same type if both are Peterson-type, 
or both are 312-type, or both are 231-type. 

Lemma 4.28. Let w, w' & Hcss(/i)'^^ . Suppose one of the following conditions hold: 

• w and w' are of the same type, or 

• w is of Peterson type and does not contain 321, and w' is either 312-type or 231-type, or 

• w is either 312-type or 231-type, and w' is of Peterson type. 

Then 

w < w' if and only if A{w) C A{w'). 

Proof. Since the lemma above shows that w < w' implies A{w) C A{w'), for all cases it suffices to show the 
reverse implication. First suppose w and w' are of the same type and A{'w) C A{'w'). An examination of 
the reduced word decompositions of the 334-type fillings given in the above discussion and an argument 
similar to that in fTTll implies w < w' . Now suppose w is of Peterson type and does not contain 321 and 
w' is either of 312 type or 231-type. Then since {1, 2} % A{w), either 1 A{w) or 2 ^ Alw). From the 
explicit reduced word decompositions of 312 or 231-type fixed points chosen above it can be seen that w' 
is Bruhat-greater than both Wjnw')\^i^ and Wji(^.w')\{2}- The claim now follows from Lemma 14.231 Finally 
suppose w is either 312-type or 231-type and w' is of Peterson type. Since A{w) C A{w') we know from 
Lemma [4.23l that Wji(w) < ''i'yi(tu') = . Lemma |4j24] shows that w < Wjk^w) so the result follows. □ 

The next step is to show that Bruhat relations between certain Hessenberg fixed points are connected to 
lengths of initial maximal consecutive substrings in the associated subsets. We need some notation. Let 
A C {1,2,. ..,71 — 1}. Recall we denote by wji the Peterson-type fixed point associated to A. For the 
purposes of this discussion we let ua (respectively va) denote the 312-type (respectively 231-type) fixed 
point with associated subset A. Thus for A ^ [1, a] for some a with 2 < a < n — 1, we have 

u\-\ „] ~ (a + la ••• 312a + 2a + 3 •■• n)~^ 

(4.27) ' ■ ' ^ ' 

= aa + la— la — 2 •■• 21a + 2a + 3 ••• n 

and 

W[ic,i = (2a + la---431a + 2a + 3- -- n)"^ 

(4.28) ' ■ ' ^ ^ 

= a + llaa— 1 ••• 32a + 2a + 3 ••• n 

in one-line notation. For general subsets 

A = [01,02] U [03,04] U • • • U [Om_i,Om], 

with oi = 1 and 02 > 2, the definitions 312-type and 231-type fixed points imply that 

(4-29) UA = U[ai,a2]W[a3,ai] ' " ' W[a„_i,a„] 

and 

(4-30) VA = Vlaua2ma3Mi] ' " ' t«[a„_i,a„]- 

Lemma 4.29. Let A, 'B be subsets o/{l, 2, . . . , n - 1} and let 

A = [01,02] U [03,04] U • • • U [o,„_i,a,„] and S = [&i,&2] U [63, M U • • • U [bm-i,b,n] 

be the respective decompositions into maximal consecutive substrings. Assume both A and 'B contain {1,2}. Let 
wa (respectively va) be the Peterson-type (respectively 231-type) fixed point corresponding to A and let its be the 
312-type fixed point corresponding to 25. Then 

Wa < u-B (respectively va < us) if and only if 71 C 23 and 62 > 02 + 1. 

Proof. We begin by recalling two basic observations about Bruhat order in Sn ■ Both follow straightforwardly 
from its definition in terms of reduced word decompositions. Suppose w, w' G Sn and assume that w and w' 
do not share any simple transpositions in their reduced word decompositions, i.e., Si < w implies Si w' 
and vice versa. Then firstly, w ■ w' < w" for w" G Sn if and only if both w < w" and w' < w" . Secondly, 
w < w' ■ w" if and only if w < w" . 
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Recall that can be written as 

(4.31) WA = U'[ai,a2] ■ W[a3,ai] ' ' " ■ 

Moreover each factor appearing in the decomposition l|4.31l l (respectively l|4.30t and l l4.29|l ) for wa (respec- 
tively va and U'z) has the property that it does not share any simple transpositions with any other factor 
appearing in the decomposition. 

Now suppose va (respectively wa) is Bruhat-less than u^. Then we know from Lemma 325] that yi C 53 
so it suffices to prove 62 > 02 + 1. From Lemma [4.91 and the definition of the Peterson type fixed points we 
know that the one-line notation for va (respectively wa) has first 02 + 1 entries 

02 + 1 1 a2 02 — 1 • • • 3 2 

(respectively 02 + 1 a2 02 — 1 • • • 3 21) while the one-line notation of U3 has first 62 + 1 entries given by 

62^2 + 1^2 - 1 ••■ 321. 

In particular 1 e D]^{va) and also 1 e DjiIwa)- By the tableau criterion, this implies that the first entry 
of the one-line notation of va and wa must be less than or equal to the first entry of that of u^. Hence 
a2 + 1 < 62 as desired. 

Conversely suppose yi C 23 and 62 > 02 + 1. Then an examination of the one-line notation of V[a-^.a2] 
(respectively W[ai,a2]) compared to that of ujf,^ and another application of the tableau criterion implies 
that W[ai,a2] < ■"[bi,h2] and W[ai^a2] < ""[61,62] • ^ particular W[a,,a2] and W[a^,a2] are al so Br uhat-less than 
u-B- Moreover since A C "B it follows that [a3,a4] U • • • U a„i] C 53 so Lemma 14.231 implies w' :~ 

^[03.04] ■ ■ ■ W[a^_j.a,„] < = ^T, ■ Si, where the last equality follows from Lemma |4!9l Since si does not 
appear in any factor of w' the general fact above implies w' < u-b- Finally since neither W[ai,a2] rior V[a-^.a2] 
share any simple transpositions with w' the other general fact above yields va < u-b, wa < u-b as desired. 

□ 

4.4. Proof of Proposition 14.31 We may now prove the upper-triangular vanishing property of 334-type 
Hessenberg Schubert classes. 

Proof of Proposition 14.31 Let w, w' G Hess(/i)'^^ and let 

A{w) = [fli, 02] U [03, 04] U • • • U [am-i,am] and A{w') = [a'l, U [03, 04] U • • • U [a'^_;^, a^] 

be the respective associated subsets decomposed into maximal consecutive substrings. By Lemmas 13.51 
and l4.4l it suffices to prove that if roU{'w) < w' , then w < w'. So suppose ro£i{w) < w' . By Lemma 14.251 this 
implies A{'w) C A{w'). By Lemma [4.28l we can conclude w <w' it one of the following hold: 

• w and w' are of the same type, or 

• w is of Peterson t5rpe that does not contain 321, and w' is either 312-type or 231-type, or 

• w is either 312-type or 231-type, and w' is of Peterson type. 

Now suppose one of the following holds: 

• w' is of Peterson type that does not contain 321 and w is not, 

• w' is 231-type and w is of Peterson type that contains 321, or 

• w' is 231-type and w is 312-t5rpe. 

In these cases, Lemma [4.27l implies that roU{w) w' so there is nothing to prove. 
It remains to discuss the cases when: 

• i(j is of Peterson type that contains 321 and w' is 312-type, or 

• w is 231-type and w' is 312-type. 

By Lemma [4.291 it suffices to show that a'2 > 02 + 1. Suppose w is of Peterson type that contains 321. In 
particular 02 > 2. From l|4.23t we know that 1 e Dj^{ro££{w)) and the first entry in the one-line notation 
of rol£{w) is 02 + 1. On the other hand l|4.27|l implies the one-line notation of w' begins with 03. So if 
roU{w) < w' then the tableau criterion implies a'2 > 02 + 1 as desired. Now suppose w is of 231-type. Again 
a2 > 2 and from l|4.25t we know roll{w) has 1 S D }i[roU{w)) and 02 + 1 as its first entry. By the same 
argument, 03 > 02 + 1 as desired. The result follows. 

□ 
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5. Combinatorial formulae for restrictions to fixed points of 334-type Hessenberg 

Schubert classes 

Our goal in this section is to give a combinatorial formula for ProU(w){'w) from which it follows as a 
corollary that it is nonzero. This proves Proposition 14.21 and hence Theorem 14.11 Although not strictly 
necessary for the proof of Proposition 112] we choose to prove the explicit formula (Proposition 15. 9l below) 
since such a formula is a first step towards a derivation of a Monk formula for 334-type Hessenberg varieties 
and because it conveys a flavor of the combinatorics embedded in the GKM theory of Hessenberg varieties 
which are larger than the Peterson varieties in |TT |. Many of our computations are analogues of those 
in im Section 5]. Our main tool is Billey's formula. We briefly recall some definitions and results (see also 
discussion in (TlJ Section 4]). 

Definition 5.1. ( Bill Definition 4.7]) Given a permutation w G Sn, an index j e {1,2,..., i{w)}, and a 
choice of reduced word decomposition b = (6i , 62 , • ■ • , ) (corresponding to the word w = si,^ - ■ ■ s^^^^^ ) 
for w, define 

(5.1) T'C?, b) := Sb^Sb2---Sbj-iitb^ ~tb^+i). 

From the definition it follows that r(j, b) is an element of i7y(pt) = Sym(t*) = C[ti, ^2, • ■ • : tn] of the 
form — tk for some £, k. These elements r{i, b) are the building blocks of Billey's formula [21 Theorem 4] 
which computes the restrictions [w) of equivariant Schubert classes CTi, at arbitrary permutations w\nSn- 

Theorem 5.2. ("Billey's formula", IT;. Theorem 4]) Let w G 5„. Fix a reduced word decomposition w ~ 
SbiSb2 ■ ■ ■ ^nd let h = (61, 62, ... , 6f(t„)) be the sequence of its indices. Let v G Sn- Then the restriction 
(Jv{w) of the Schubert class ay at the T -fixed point w is given by 

(5-2) (Tv{w) = ^r(ji,b)r(j2,b) • • •r(j£(^),b) 

where the sum is taken over subwords Sb^^ Sbj^ ■ ■ ■ Sb-^^ ^ ofh that are reduced words for v. 

We record the following fact, used in the proof below, which follows straightforwardly from the Billey 
formula. 

Fact 5.3. Suppose v,w G S„ with v < w in Bruhat order. Suppose there exists a decomposition w ~ w' ■ w" 
for w',w" G Sn where v < w' and, for all simple transpositions s; such that Si < v, we have Si ^ w". Then 

Following terminology in |fTT|. we refer to an individual summand of the expression in the right hand 
side of l|5.2|l , corresponding to a single reduced subword v = Sf,^^ Sb^^ ■ ■ ■ Sb^^^ ^ of w, as a summand in 
Billey's formula. In order to derive formulas for py (w) where py is a Hessenberg Schubert class, we use the 
linear projection 7751 : t* — > Lic(5^)* dual to the inclusion of our circle subgroup S^ into T given by l|2.2|l . 
More specifically, since the diagram | |4.6|I commutes, we have 

(5.3) py{w) = ^7r5i(r(ji,b))7r5i(r(j2,b)) • ■ •7r5i(r(jf(„),b)). 

We refer to the right hand side of the above equality as Billey's formula iorpy{w). Recall 7151 {t£ — tk+i) — 
{k + I — i)t for a positive root — tk+i HH Section 5]. 
We also use the following. 

Definition 5.4. ( (111 Definition 5.4]) Fix C {1, 2, . . . , n - 1}. Define : ^1 ^ yi by 

3^yi (j) = the maximal element in the maximal consecutive substring of A containing j. 

Definition 5.5. ( (111 Definition 5.5]) Fix C {1, 2, . . . , n - 1}. Define 7a : A ^ Ahy 

7ji{j) = the minimal element in the maximal consecutive substring of A containing j. 

We proceed to some preliminary computations. Let b = . . . , 6£(^)) be a reduced word decomposition 
w = Sbi S62 ' ' ■ •Sfc<!(„) of w and let i be an index appearing in b, i.e. bg ^ i for some 1 < £ < i{w). Our first 
computation. Lemma [5.61 gives an expression for 7751 (r(^, b)) which shows in particular that the value of 
7751 {r{t, b)) depends only on the value of the index bg = i and not on its location £ in the word b. Note that 
if 1; = Si then the summands in Billey's formula for py{w) = Ps^ [w] are precisely equal to r{£, b) for each 
£ such that bg = i. Thus an equivalent formulation of the claim is that the summands in Billey's formula 
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for ps- {w) are all equal. This is analogous to a result in the Peterson case [11, Lemma 5.2] except that in 
our situation, the form of the formulas depend on the index i as well as on the type of the fixed point w in 
question. 

Lemma 5.6. Let w e Hess(ft-)"^^ and let h = . . . , b^i^^^) he the reduced word decomposition of w chosen in 
Section^ Let A{w) = [ai, 02] U [03, 04] U • • • U [am_i, a„] he the associated suhset ofw decomposed into maximal 
consecutive suhstrings. Let i e {1, 2, . . . , n — 1}. 

(1) Ifi ^ A{w), then each summand in Billey's formula for ps- {w) is 0. In particular, ps^ (w) = 0. 

(2) Suppose i G A{w) and suppose one of the following conditions hold: 

• w is of Peterson type, or 

• w is 312-type, or 

• w is 231-type and i ^ [ai, 02]. 

Then each summand in Billey's formula for pg. {w) is equal to 

(3) Suppose w is 231-type and i = 1. Then each summand in Billey's formula for ps^iw) is equal to 

02* = J{yl(u,)(l)t. 

(4) Suppose w if 231-type and i e [2, 02]. Then each summand in Billey's formula for ps- (w) is equal to 

(«-T/i(«,)W)i=(*-l)t- 

Proof. If i does not occur in A{w) then each summand is by Billey's formula for (Ts^{w), since Si -jit w 
and thus never appears in the reduced word decomposition of w. For the next claim, the fact that each 
summand is equal to {i — 'Ta(iu) (i) + l)t for the listed cases follows from examination of the chosen reduced 
word decompositions of w and an argument identical to that in flU . Thus it remains to check the cases in 
which the summand differs from the case of Peterson varieties. First suppose w is 231-type and that i = 1. 
From the choice of explicit reduced word decomposition for such w given in l|4.18t and Billey's formula, it 
follows that each summand in Billey's formula for CTsj (w) is equal to 

(5.4) 

r2{r3r2) ■ ■ ■ (ra2_ira2_2 • ■ ■ r3r2){ra2ra2-i • • • ^2 (ti - t2)) = r2{rzr2) ■ ■ ■ [ra2^ira2-2 ■ • •''3'"2)(ii - ^02+1) 

= tl — ta2 + l 

since the reflection rj switches tj and tj+i. Hence we have Ps^ {w) — 7151 (ti — ia2+i) = {^2 + 1 — l)t = a2t = 
^A{w) (1)^- Now suppose w is 231-type and i G [2, 02]. The factor in the reduced word decomposition | |4.18|I 
corresponding to [1 = ai, 02] is equal to W[2.a2] ' ^i- By Fact l5.3l for i > 1 the presence of the extra si does 
not affect the Billey computation, so each summand is equal to that for the Peterson type fixed point ^[2,02] 
and hence is equal to 

as desired. □ 
Our next lemma concerns the summands in Billey's formula for ProU{w) (w) for w G Hess(/i)'^ . 

Lemma 5.7. Let w G Hess(/i)'^\ 

• Suppose w is 312-type or w is of Peterson type that does not contain 321. Then each summand for Billey's 
formula for Proee{w){'w) is equal to 

Suppose w is 231-type. Then each summand for Billey's formula for p^oU[iu){w) is equal to 

^AMii)-[ n ■ n (()»)+ 

V i=2 J V'e-^("')\P^(-)(i)>^-«(™)(i)l / 
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Suppose w is of Peterson type that contains 321. Then each summandfor Billey's formula for Proee{w){w) is 
equal to 

Proof. Before considering the separate cases we make a general observation. By Lemma [5. 61 and the discus- 
sion before Lemma [5.6l vye know that the summands in Billey's formula for p^. {w) for i G A{w) are exactly 
the terms 7r5i {r{£, b)) for £ such that be = i. Suppose in addition that w 6 Hess(ft,)'^^ is such that roM{w) 
contains at most one simple transposition Si for each i e {1, 2, . . . , n — 1}, i.e., roi£{w) = Si^s^^ ■ ■ ■ Si^^^^^^^^^j 
is a reduced word for rol£{w) where all ik are distinct for 1 < fc < l{rol£{w)). This implies that any subword 
of a reduced word decomposition b of w which is a reduced word of rot£{w) also must contain precisely 
one Si^ for each 1 < fc < t{rol£{w)). From Billey's formula l|5.3b for ProU{w){w) we know that a summand is 
of the form 

(5-5) TTgi (r(ji , b))7rsi (r^z, b)) • • • tt^i (r(jf , b)) 

where Sfe.^ Sf,^.^ ' ' ' is a reduced word of r-o^£(w). Since {6^1, foj^^^^^^^j, } = {ii,i2, ■ ■ ■ ,H{roU{w))} 
for each such summand the quantity (|5.51 is equal to 

l.(roU{w)) 

(5.6) n p^^»- 

k=l 

We now take cases. Suppose w is not a Peterson-type that contains 321. Then from the explicit descrip- 
tions of ro££{w) given in Section|4]it follows that ro££{w) contains in its reduced word a single Sj for each 
i S A{w). Thus we are in the situation described in the above paragraph and the claims follow from the 
computations given in Lemma [5. 61 

Suppose w is Peterson type and contains 321. Let b be the standard reduced word decomposition 
(cf. (|4.131 and ll4.15l l) of w. We claim that the only reduced word decompositions of rol£{w) that occur 
as a subword of b are those which contain two si's, only one S2, and precisely one Sj^ for all other ji. In- 
deed let A{w) — [ai, 02] U [03, 04] U ■ • • U [cm-i, Om] be the decomposition of A{'w) into maximal consecutive 
substrings. Recall 02 > 2 and ai = 1 in this case. The rolldown ro££{w) is 

(Sa„So„.-l • ■ • So„_i) • ■ • (S04S04-I • • • Saf) ' {Sa^Sa^-i ■ ■ ■ S1S2S1) 

and w is 

Let £ > 1. There is only one reduced word decomposition of the factor Sa^^^Saf^i-i • • - Sa^ in ro££{w), 
so it remains to analyze the subwords of ^[01,02] which are reduced words of SaaSaa-i ' • ■ siS2Si- Let b 
denote the standard reduced word of W[ai,a2]- Note that another valid reduced word of Sq^Sq^^i ■ • • siS2Si 
is 502*02-1 ■ ■ ■ s2SiS2- Since S2 does not commute with si, the rightmost S2 in the word Sq^Sqj-i • • • S2S1S2 
must appear to the right of the s^^; in particular, there are two S2's to the right of the in this word. 
Since there is only one §2 appearing to the right of the Sa2 in b we conclude that the reduced word of 
502*02-1 ■ ■ ■ S1S2S1 containing two copies of S2 never appears as a subword of b. Hence the only subwords 
of b contributing to summands in Billey's formula for Proa{w){w) contain two si's and one S2, as claimed. 
Now since ji = 1 and j2 = 2 and Ps^ (w) = (1 — Tyij^u) (1) + l)t ^ {1 — 1 + l)t = t by Lemma [5.61 the claim 
follows. □ 

We have just seen that all summands in Billey's formula for ProU{w){w) are equal for all fixed points 
w G Hcss(/i)'^ . In order to finish the computation we must now compute the number of summands which 
occur. 

Lemma 5.8. Let w e Hess(/i)'^\ 

• Suppose w is of Peterson type that contains 321. Then the number of summands in Billey's formula for 

ProU(w){w) is J{yi(u,)(l) - 1. 

• Suppose w is of Peterson type that does not contain 321. Then the number of summands in Billey's formula 
for ProU(w){w) is 1. 

• Suppose w is 312-type. Then the number of summands in Billey's formida for ProU{iu) (w) is J-(a{iv) (1) — 1- 
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• Suppose w is 231-type. Then the number ofsummands in Billey's formula for p^oU{w){w) is 1. 

Proof. We consider each case in turn. Suppose w is of Peterson type that contains 321. Let A{w) = [ai, 02] U 
[03, 04] U • ■ • U [a„i_i, a,„] be the decomposition of A{w) into maximal consecutive substrings. Recall 02 > 2 
and oi = 1 in this case. The roUdown ro££{w) is 

and w is 

Let £ > 1. As observed in the proof of Lemma 15.71 there is only one reduced word decomposition of the 
factor Sa£+iSa{+i-i ■ ■ ■ Sai in ro££{w). Moreover by examination it is evident that it appears only once in 
the standard reduced word decomposition of the corresponding W[ag^at+i] factor in w. Hence in order to 
coimt the number of ways ro£t{w) appears in w it suffices to count the number of subwords of the standard 
reduced word decomposition b of w^ai,a2] which are reduced subwords of Sa^Sa-^-i ■ ■ • S1S2S1. We already 
saw in the proof of Lemma [5.71 that the reduced word SqjSq^.i ■ ■ ■ S2S1S2 never appears in b. On the other 
hand since si commutes with any Sk with A: > 3, another reduced word decomposition of Sa2-i ■ ■ ■ S1S2S1 
is siSa2Sa2-i ' ' ' S2Si- From examination of b it can be seen that the word siSa^Saa-i • • ■ S2S1 appears as a 
subword in the standard reduced word of ^[01,02] precisely 02 — 1 = ^Kyi(„,) (1) — 1 times and that these are 
the only subwords of b which equal SaaSaa-i • • • siS2Si- The claim follows. 

Suppose w is of Peterson type that does not contain 321. Then the rolldown ro££{w) is the Peterson case 
roUdown so the claim follows from explicit examination of the standard reduced word of w (alternatively 
from fTTl Fact 4.5]). 

Suppose w is 312-type. Then the rolldown ro££(w) is of the form 

ro££{w) = (Sa,„Sa„-l ■ • • Sa„_i) ' ' ' (Sq^Sq^-I ■ ' ■ ) ■ {Sa^Sa^-l ' ' ' S1S2) 

from Lemma r4.20l By an argument similar to the case of Peterson type that contains 321 it suffices to analyze 
only the factors in both ro££{w) and w corresponding to the initial maximal consecutive substring [ai, 02]. 
As above we have 

Sa2^a-2-l ' ' ■ ^1^2 SiSa2Sa2-l ' ' ' S2 

and again it follows from examination of the standard reduced word of ujaj aj] that siSaaSaa-i ■ ■ ■ appears 
precisely a2 — 1 = ^Kyi(^)(l) — 1 times. 

Finally suppose w is 231-type. Then the rolldown ro££{w) coincides with the Peterson case rolldown of 
wa(w) and the claim follows from examination of the reduced word decomposition (|4.18t . □ 

The following is immediate from Lemmas 15 . 71 and I5l8l 

Proposition 5.9. Let w e Hcss(/i)'^\ 

• Suppose w is of Peterson type that contains 321. Then 

ProUMH = (M^(^)(l) - 1) I n - + 1) j • il^^'^^l+l. 

• Suppose w is of Peterson type that does not contain 321. Then 

ProU(u,){w) - n " 7A(n,){i) + 1) t'^^"^'. 

• Suppose w is of type 312. Then 

ProUMiw) = i^AMil) - 1) n " '^AMii) + 1) • i'^'"^'. 

\iG.A(i«) / 

• Suppose w is of type 231. Then 

ProuMH = ^am{i) • n - 1) • n - ^^(-)(()*) + 1) i''^^"^'- 

\ «=2 / \»eA(«;)\[T^(„)(l),M^(„)(l)] / 
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The proofs of the main results are now immediate. 

Proof of Proposition \4.2\ Let w e Hess(/i)"^\ From the explicit formulas given in Proposition 15.91 it follows 
that ProU{w) {w) ^ for all possible types of fixed points w. □ 

Proof of Theorem |4I7] Since both | |4.3l l and (|4.4l l are satisfied for all w,w' e Hess(/i)'^^ by Propositions 14.31 
and 14 . 2 1 respectively, the result follows. □ 

6. Open questions 

This manuscript raises more questions than it answers. We close by mentioning some of them. 

Question 6.1. For n > 4, Theorem Wl\ shows that for the case when iV : C" — > C" is the principal nilpotent 
operator and h is the 3M-type Hessenberg function, the dimension pair algorithm produces a set ofHessenberg Schu- 
bert classes {pro«(u))}„,gHcss(Af /i)^^ wMch are poset-upper-triangidar and hence form a Hgi{pt)-module basis for 
H*,(Hcss{N,h)). 

(1) What are other examples of N and h such that the conclusion of Lemma \3. 4\ holds (cf. Remark [3^ ? 

(2) What are other examples of N and hfor which the dimension pair algorithm produces a successful outcome 
ofBetti poset pinball which is also poset-upper-triangular? Are there necessary and sufficient conditions on 
N and h that guarantee poset-upper-triangularity? 

(3) What are other examples of N and hfor which the dimension pair algorithm produces a successful outcome of 
Betti poset pinball which corresponds to a linearly independent set of classes and hence a module basis? Are 
there necessary and sufficient conditions on N and h that guarantee this? 

Question 6.2. In iHTlf the explicit module basis consisting of Peterson Schubert classes is used to derive a manifestly 
positive Monk formula in the S^-equivariant cohomology of Peterson varieties. Preliminary investigation suggests 
that an analogous Monk formula for the iZA-type Hessenberg varieties, using the module basis ofHessenberg Schubert 
classes derived in this manuscript, woidd be computationally much more complex. Thus, we may ask the following. 

(1) Does there exist a combinatorially elegant or computationally effective Monk formula for the 334-type Hes- 
senberg varieties? 

(2) Can such a Monk formula be further generalized to a larger family of regular nilpotent Hessenberg varieties? 
For instance, can our techniques be generalized to give new insights to the equivariant Schubert calculus of 
the full flag variety 3'£ags{C"') (which is an example of a regular nilpotent Hessenberg variety)? 

(3) In ^ the Monk formula for Peterson varieties is used to derive a Giambelli formula. Does there also exist a 
combinatorially elegant and/or computationally effective Giambelli formula for other cases of regular nilpotent 
Hessenberg varieties? 
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